Dukes Academy Analysis

Introduction

It is known that supplemental instruction is effective in reducing drop, failure, and withdrawal rates in traditionally challenging courses such as biology and chemistry. However, there is a disparity in student access to traditional on-campus supplemental instruction programs such as tutoring centers and TA-facilitated review sessions. To date, there are limited data describing the effects of student-made online supplemental instruction (OSI) on grades, and fewer still addressing how online options for supplemental instruction mitigate accessibility rifts that exists between students. With this shortage of data comes an exceptional opportunity to gain foundational knowledge about how student-made OSI materials impact grades, learning outcomes, and student’s accessibility to resources for success. High-quality student-created supplemental instruction resources have the potential to change the way science courses are experienced at JMU and other universities. Our project also offers an exciting opportunity for undergraduates to work collaboratively and creatively with other undergraduates and faculty to produce resources that contribute to the learning community of JMU.

Project Description

Roughly a year ago, our group of three tutors/TAs noticed that there were some students that desired to come to our organic chemistry tutoring sessions and the tutoring center but were restricted by various situations. Students either did not have schedules conducive to attending or were unable to travel to campus to receive extra help. Since it is known that supplemental instruction is correlated with higher pass rates in challenging courses, we felt that these students were at a disadvantage when compared to their peers who could regularly attend. We decided to create resources that could be accessed online, by everyone, in order to increase the accessibility to supplemental instruction resources at JMU.

We were originally inspired by YouTube channels that had been useful to us in our own challenging courses in the past. Channels such as Kahn Academy, Armando Hasudigan, AK Lecutres, The Organic Chemistry Tutor, ect. featured content creators offering unique ways of explaining difficult concepts with the added luxury of being able to pause, rewind, slow down, or speed up at one’s discretion. We wanted to create a channel like Kahn Academy which featured tutorial videos made by students who had been successful with a certain professor in the past. The students using the content would then learn first-hand how to be successful with Professor “X” with pro tips from a peer that had already aced X’s class.

We knew from our combined experience as tutors and TAs that chemistry students love practice problems. The most frequent request from students on feedback surveys was always: “please bring more practice problems.” So, we decided that instead of re-lecturing or going over class notes in our videos, we would explain solutions to practice problems that we created with the foundational concepts (and the professor’s unique curriculum) in mind.

We wanted to give students more than just practice tests and answer keys. We wanted to give detailed explanations as to why the answers were correct or incorrect, similar to the experience you would receive during a tutoring session. After all, our mission was to increase accessibility to tutoring resources. Drawing from our early inspirations, we decided to utilize the widely popular Khan-Academy style of “screen-casting.” That is, we recorded ourselves working through problems on an iPad, explaining the thought processes and offering course specific tips along the way. We would ultimately make five tests filled with comprehensive practice questions- each test is accompanied by an answer key and explanation videos for each question. Below is an example of our content.

Example Problems/Answer Key

Example Video Explanation

What originally began as a small passion project eventually became a very rewarding task. As we put in more and more of our time, we took pride in the quality and effect of our resources. After several revisions we were proud to put our name on the final product. However, the most important question remained unanswered: does this novel resource help students learn organic chemistry? In order to move towards answering that, we collected data in the spring of 2019 on an organic chemistry I class. We collected very basic data for this exploratory analysis: how many times did a student click on our content, and what was their test score?

The Sample

Although a seemingly small detail, the fact that we collected data in the spring of 2019 on an Organic Chemistry I class raises questions regarding the representativeness of our sample. Generally, organic chemistry I is taken in the fall semester by students interested in M.D., D.D.S, and Veterinary programs, as well as biology majors. The pre-health and biology advising staff usually recommends this schedule to their students. Thus, spring sections tend to contain less of these individuals and more non-traditional students who may be re-taking the course.

It is not to say that resources are not intended for everyone, however the results of this analysis may not be representative of the typical composition of an organic chemistry 1 section. Adding to this effect, the professor who taught our sample cohort had never taught in the spring before, and was forced to adapt his M/W/F teaching schedule to a T/Th schedule. It is important for us to express these points because of the effect they likely have on our data and the significance of our preliminary results.

The grades from our sample (a total of 78 students)are shown below. We note that these grades are far lower than normal-not only for this instructor but also when compared to other chemistry faculty. These findings add validity to our skepticism towards the representativeness of this sample.

Grade Distributions By Exam

P.S. - Most of the figures in this presentation are fully interactive. Click a box once for the raw distribution, and twice to reset them.

Video Use

Before we explore the relationship between grades and the number of times the students accessed the resources, we want to demonstrate that the resources were actually used. Below we have included a figure showing the number of clicks that our resources got, relative to the date. We have included exam dates for reference.

Resource Use vs. Time

Clearly, the resources were used. The data indicate that the bulk of the use was almost exclusively right before each exam. We believe this trend is explainable to a large extend by our data collection methods, the way the resources were released, and student awareness of the resources’ availability:

  • The nature of Canvas data output makes it very difficult to count multiple views of one resource. If a student watched one video 5 times, it is only recorded as a single “view” in these metrics. This means that the number actual views is likely even higher than what is depicted here.
  • Seeing as we were still in the process of finalizing some of these videos during the spring semester, we were often unable to upload the videos more than two weeks earlier than their respective tests dates (although we always uploaded the resources at least one week before the respective test). This is a likely contributing factor to the periods of inactivity observed between tests.
  • The only way we were able to announce the upload of these videos were through Canvas announcements. Ideally, we would have the teacher or TA make more regular announcements and reminders in person during class time.

Interestingly, there is a large, variable proportion of individuals who do not use the resource at all (0 views). These non-users are shown below by each test, as well as over the course of the entire year.

Proportion of Non-Users

Interestingly, the largest proportion of non-users was observed during before Exam 4 and before the Final Exam. This may be explained by the relatively quick turnaround between these exams, as well as the stress and time constraints related to finals week. However, we are excited to learn that 90 percent of students used the resources at least once.

Our take-away regarding total recorded resource use was this: despite less than ideal conditions there were still hundreds of resource views for each practice set, and this number is likely deflated. After finding that our resources were actually used by students, we wanted to look closer on the effect our resources had on grades.

Correlation: Total Clicks vs. Total Grade

Below is a regression analysis comparing total clicks vs. grade for students in our sample.

Regression

There are multiple aspects of this graph to interpret. Most notable are the R2 value and the p-value in red. Each will be addressed individually.

  • R2 Value: This metric is used to assess the fit of the model, and is derived from the correlation coefficient r. A good model is generally characterized as having a high R2 value, but there is more to the story. By definition, an R2 value of 0.052 means that 5.2% of the variability in the sample’s grades could be explained by how many times students viewed our resources. Our R value seems reasonable when we consider all of the factors that could cause variability in the grades of a class (an individual’s personal motivation, their office hour attendance, whether they do the assigned homework, etc.). We would not expect our resources to account for an excessively large portion of grade variability, and so this result makes sense.

  • p-value: This relates to the hypothesis that there is NO relationship between clicks and grades. Using a cutoff of 0.05, these results are considered statistically significant. In other words, this analysis revealed that there is a significant relationship between clicks and final grade, although due to the nature of the study causation cannot be established. This means that there is no way of telling whether or not more clicks caused individuals to have a higher grade, or if the individuals who got higher grades were simply predisposed to use the resource more.

To us, these results are very promising. However, the results from this cohort is not the end-all-be-all in whether or not our resources are effective. We need a larger, more representative sample and better data collection methods before we can conclusively say that the resources do or do not have a significant effect on grades. With this in mind, there are still additional lenses through which we can view these data.

Grades: High Use Vs. Low Use

One thing we were interested in was the average grade of students who were “low users” of our resource versus who were “high users”. Here, we set arbitrary cutoffs on the amount of total views the students had. If a student’s views were above this cutoff, they were deemed a “high user”, and if below, deemed a “low user”. We then compared the average grades of all high and low users to determine whether or not they were significantly different.

Given the large skew in our data, we chose the arbitrary cutoff to be the median: 11 total views.

Determining the Cutoff

When we conducted our analysis on these bins of users, we initially found that there was no statistically significant difference (p = 0.314) between high and low users, although this may be attributed to high levels of variation in the data (shown below).

Grades of High vs. Low Users

Looking closer, we can see that the three individuals who received the highest grade fell into the “low” category. These students are likely very bright and do not need additional help to succeed. Similarly, we see that some of the very lowest overall grades also belong to the “low” group. These could be the students who weren’t motivated, skipped lectures, or just didn’t study (notably, a few students in this sample skipped not one, but TWO of the five exams).

When we eliminate these extremes from our sample, we can get a better look at the effect our resources had on the typical student’s grade. We chopped off these extremes and looked only at those students who finished with a final grade within two standard deviations on either side of the mean. When we filter out the extremes, we see that the difference between grades of low and high users is more pronounced, but still not significant (p = 0.112) .

Grades of High vs. Low Users Reconsidered

Conclusion

The methods used in this preliminary inquiry are less than ideal for generating representative data and we hope to expand our studies to a larger sample that is more diverse and representative. However, we believe that these early results provide reasonable justification for future examination.

So far, we have only been able to collect and interpret click and grade data. In the future we hope to find collaborators that are well-versed in assessment and measurement so that we can obtain more survey-based, subjective data about the participants and their learning experience. A major goal is to be able to explore questions about how accessibility to supplemental instruction affects students’ ability to succeed in challenging courses.

Currently, we are analyzing data from our most recent cohort that we believe will be more representative of the target population than these data. We are also working to expand our content library and findings to general chemistry courses, to ensure that our results translate across different courses. We hope to expand this model to other institutions in order to broaded the scope of students that we are able to help succeed in these challenging courses. By fostering connections between institutions, we hope to pursue grant assistance by institutions such as 4VA and others to allow the project to flourish to its full potential.

Shane Chambers

7/14/2019